# Plan: broker/tests/failure_injection/ Harness
## Goal
Scaffold a wire-level IBKR mock that deterministically injects 9 failure modes, with both:
1. Pure Rust integration tests for fast, deterministic testing of individual failure modes
2. Python-driven scenario runner for complex multi-step scenario testing
## Phase 1: Directory Structure & Core Mock (verify: dir created, compiles)
- [ ] Create `broker/tests/failure_injection/` directory
- [ ] Create `broker/tests/failure_injection/mod.rs` (test module)
- [ ] Create `broker/tests/failure_injection/mock_tws.rs` (wire-level TWS mock)
- [ ] Create `broker/tests/failure_injection/scenarios/` directory (Python scenarios)
- [ ] Add `broker/tests/failure_injection/README.md` (usage docs)
## Phase 2: Wire-Level TWS Mock (verify: mock compiles, basic test passes)
- [ ] Design `MockTws` struct that mirrors TWS API surface:
- Callback ordering simulation
- Sequence number tracking
- Partial-fill semantics
- Deterministic failure injection API
- [ ] Implement failure injection trait/module:
- `FailureMode` enum (F1-F9 variants)
- `inject_failure(mode)` method on MockTws
- Configurable failure timing (pre-submit, post-submit, mid-fill, etc.)
- [ ] Implement basic MockTws connectivity (connect/disconnect)
- [ ] Write smoke test: MockTws connects, submits order, receives normal callback
## Phase 3: Rust Integration Tests (verify: 9 test functions compile and run)
- [ ] F1: Test duplicate order-status callback injection
- [ ] F2: Test cancel reject race with fill
- [ ] F3: Test partial fill followed by disconnect
- [ ] F4: Test stale market data detection
- [ ] F5: Test clock skew detection
- [ ] F6: Test TWS reconnect drill
- [ ] F7: Test cron double-fire idempotency
- [ ] F8: Test kill switch subcommand
- [ ] F9: Test process crash + warm restart
## Phase 4: Python Scenario Runner (verify: Python script runs, executes scenario)
- [ ] Create `broker/tests/failure_injection/scenarios/runner.py`
- [ ] Implement scenario DSL (YAML or JSON) for declaring test sequences
- [ ] Add FFI bindings or CLI interface to drive Rust MockTws from Python
- [ ] Write 1-2 example complex scenarios (e.g., F3+F6 combined)
- [ ] Add scenario runner to CI
## Phase 5: Documentation & Integration (verify: README complete, CI passes)
- [ ] Document MockTws API in `broker/tests/failure_injection/README.md`
- [ ] Document scenario DSL in README
- [ ] Add failure injection tests to CI workflow
- [ ] Run `cargo test --package broker` to verify all tests pass